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Abstract 



We report the first observation of B — > rj'X transitions with high momen- 
tum rj' mesons. We observe 39.0 ± 11.6 B decay events with 2.0 < p„i < 2.7 
GeV/c, the high momentum region where background from b — > c processes 
is suppressed. We discuss the physical interpretation of the signal, including 
the possibility that it is due to b — > sg* transitions. Given that interpreta- 
tion, we find B{B -» rfX a ) = (6.2 ± 1.6(stat) ± 1.3(sys)ti° 5 (bkg)) x 10" 4 for 
2.0 < jy < 2.7 GeV/c. 
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Decays of the type b — ► sg*, gluonic penguins, are likely to be important in future studies of 
CP violation. Gluonic penguin modes will be used to search for direct CP violation, and could 
complicate the interpretation of some measurements of indirect CP violation. CLEO has reported 
the observation of signals in B° — > K~ir + [|J and B~ — > K~rj pj, exclusive modes which are 
expected to be dominated by the gluonic penguin amplitude. The inclusive decay B — ► rj'X, where 
the collection of particles X contains a single s quark, is another signature of b — > sg* (followed by 
g* — > uu, dd or ss). Here we report the observation of the inclusive process B — > rj'X and examine 
these data for evidence of b — > sg*. 

The data sample used in this analysis was collected with the CLEO II detector at the Cornell 
Electron Storage Ring. This detector is designed to measure charged particles and photons with 
high efficiency and precision ||. The data sample has an integrated luminosity of 3.1 fb _1 and 
contains 3.3 x 10 6 BB pairs. Another data sample with an integrated luminosity of 1.6 fb was 
taken at an energy 60 MeV below the T(4S) resonance and is used to subtract the continuum 
background. 

To isolate the signal and reduce the large background from continuum production of rj mesons, 
we apply the B reconstruction technique that was previously used to isolate an inclusive signal for 
b —* «7 pi. This technique selects B — > 77 X events in which the decay products of X include a 
charged kaon candidate in order to enhance the probability of observing b — > sg* . 

We search for r/'s with laboratory momenta in the range 2.0 < p < 2.7 GeV/c, beyond the 
kinematic limit for most b — > c decays. This range corresponds to a region in X mass from zero 
to 2.5 GeV. In this momentum range we should be sensitive to b — ► sg*. However, b — > u decays 
with rj mesons, such as B~ — ► ir~rj, and color-suppressed b — » c decays, such as B° — * D°rj', also 
populate this interval. Methods for discriminating among these possibilities will be discussed later. 

Events are selected using standard criteria for hadronic final states, and we consider well- 
reconstructed tracks and photons. Candidate rj mesons are then reconstructed in the rj — > r/7r + 7r~, 
rj — > 77 mode. For each r\ candidate, the 77 invariant mass must be within 30 MeV of the nominal 
r] mass. The rj candidate is fit kinematically to the rj mass and is then combined with the charged 
pions to form the rj candidate. 

We then form combinations of a charged kaon, an rj and n pions where n < 4 (at most 
one of these pions is allowed to be neutral). Eight decay modes and their charge conjugates are 
considered: B~ -> K~rf, B° -> K~jj^, B~ -» K-rj'ir , B~ -► i^r/V^, B° -> K-rfir+w , 
B° — > K~r/' 7r + ir~7r + , B~ — > K^tj'tt^tt^tt and B° — > K~rj' ir + 7r~ir + 7r . For the charged kaon 
candidate we require that dE/dx be within three standard deviations of the expected value. We 
then require that these combinations be consistent in beam-constrained mass {Mb) and energy 
difference (AE = E ^ serve( i — Eb eam ) with a B meson. (Here Mb denotes the invariant mass with 
the energy constrained to the beam energy.) We require \AE\ < 0.1 GeV and Mb > 5.275 GeV. 
In case of ambiguous hypotheses we choose the best candidate in each event on the basis of a x 2 
formed from Mb and AE. 

Following reference Q, we suppress the jet-like continuum relative to the spherical BB events 
with requirements on R2, (the ratio of second to zeroth Fox- Wolfram moments) and 6s (the angle 
between the sphericity axis of the B candidate and the sphericity axis of the remainder of the event). 
i?2 is large for jet-like events and small for spherical events. The variable cos 6$ is isotropic for 
signal events and peaks near cos 8s = ±1 for continuum. We require R2 < 0.45 and | cos^sl < 0.6. 

The rjTT + ir~ mass spectrum in the high momentum window 2.0 < p^i < 2.7 GeV/c for the 
on-resonance and off-resonance samples is shown in Fig. 1. A fit to the rj peak finds 50.7 ± 8.6 
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FIG. 1. The distribution of rjTT + ir mass for (a) on-resonance data and (b) below-resonance data 



events on the T(4S) resonance and 6.1 ± 4.1 off resonance (unsealed). After accounting for the 
differences in luminosity of the two samples, this gives an excess of 39.0 ± 11.6 events. Other ways 
of determining the yield give consistent results. We estimate a systematic error of 3% from the 
uncertainty in the fitting procedure. 

We now study the invariant mass spectrum M{X S ) of the particles recoiling against the rj . 
The continuum-subtracted M(X S ) distribution is shown in Fig. 2 and tabulated in Table |. The 
peak at the kaon mass due to the exclusive decay mode B~ — > rj' K~ 0, accounts for about 10% 
of the inclusive yield. There is no significant excess in the K* mass region. The remainder of the 
yield comes from events with X s mass near or above charm threshold. Five sources can contribute 
to this distribution: rj from secondary decays b — ► c, c — > rj'; color-allowed b — » c; color-suppressed 
b — > c; b — > u; and b — ► sg*. 



TABLE I. Yield in M{X S ) bins for B — > rfX s . The off-resonance yields must be scaled by 
1.908 to account for differences in energy and luminosity. The detection efficiency drops as M(X S ) 
approaches 2.5 GeV because of the rj' momentum cut. 



M(X S ) (GeV) 


N (on) 


N(off) 


Yield 


0.4 < M(X S ) < 0.6 


4 





4±2 


0.6 < M(X S ) < 1.2 


2.7 ±2.1 


0.6 ± 1.1 


1.6 ±2.9 


1.2 < M(X S ) < 1.8 


18.0 ±4.9 


6.6 ±3.2 


5.4 ±7.6 


1.8 < M(X S ) < 2.5 


26.0 ±6.4 


-0.8 ±2.3 


27.5 ±7.8 



Secondary decays have been reliably simulated with the Monte Carlo program. These include 
processes such as B° — > D + it~, D + — ► ri'ir + and B — > D~D + , D~ — > ir~r]'. The yield from 
secondary sources is thus estimated to be less than 0.2 events. 

We have also considered the possibility of color- allowed b — > c backgrounds such as B — > Drj'ir. 
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FIG. 2. The continuum-subtracted M{X S ) distribution (points with error bars) with expected 
M{X S ) distributions for a mixture of two-body b — > sqq (solid) and three-body b — > sg* with 

— * grj' simulated using JETSET (dashed). The data points have been corrected for the M(X S ) 
dependent efficiency. Each simulation has been normalized to the data yield. 

We have searched for this decay in a lower rj momentum range, modeling the decay with 3-body 
phase space. This search gives an upper limit of B{B —* Dr/'ir) < 1.3 x 1CP 3 , corresponding to 
a background of fewer than 1.4 events in the signal region. Thus, neither secondary decays nor 
color-allowed b — > c decays are a significant source of the high momentum if signal. 

We next consider b — > u modes. First, we check for the presence of an s quark in the final 
state by forming a x 2 difference based on AE and the resolution- normalized dE/dx residual for 
the candidate kaon. The AE distribution for b — > u modes is shifted above that for b — > s modes 
because the kaon mass is attributed to a pion. A fit to this x 2 difference, using B — » rj'ir, rj' p, rj'ai 
for the b — > u contribution and a model of b — > sg* for the b — > s contribution, gives the yields 
f(b -> sg*) = 82 ± 20% and f(b -»• u) = 18 ± 20%. 

Further information onb^u comes from the M{X S ) distribution. The dominant b — > u modes 
with an rf are expected to be B — > rfir,rfp and Vj'a\. These modes have X s mass below 1.8 GeV, 
where we see no strong evidence for a signal. The theoretical expectation for b — ► u, []|] or summing 
the exclusive modes which may contribute ||, is (3.5 — 7.0)% of the signal yield. The contribution 
with M(X S ) > 1.8 GeV and 2.0 < p v > < 2.7 GeV is likely to be much smaller. 

Experimental searches for the color-suppressed b — > c modes B° —> D(*'t/ || [^|, while showing 
no evidence for them, place unrestrictive upper limits. Theoretical expectations for the branching 
fractions for these modes are in the range (1.5 — 6.0) x 10~ 5 jj]], implying a yield of high momentum 
rj' of (2.1 - 8.6)% of the observed yield. 

To search for these modes in the data, we examine the M{X S ) distribution for neutral modes. 
The mode B° — > D r/ has a spike at the D° mass, while that for B° — > D*°r]' has a broader peak 
at the D*° mass as shown in Fig. 3. This distribution limits the contribution of B° — > D°tj' to 15% 
of the signal. 

Information about B° — > D*°T]' comes from the mass distribution obtained by removing a single 
pion (charged or neutral) from the X s system, such that the remaining particles are consistent with 
coming from a D° decay. B° — > D* rj' peaks sharply at the D mass in this distribution as shown 
in Fig. ||. The absence of such a peak in the data limits B° — > D* 0/ q' to 26% of the signal. The 
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FIG. 3. The continuum-subtracted distribution of X s mass for neutral modes with the expec- 
tations from B° — > D°r]' (solid) and B° — » D*°rj' (dashed) superimposed. The simulations have 
been normalized to the data yield. 



limits on D°r]' and D*°r]' constrain the total contribution of color suppressed decay to be less than 
41% of the signal. 

We have also tried to describe the data in Fig. 2 with a combination of B° — > D°r)', B° —* D*°r)', 
and _B° — > D**(2.2)r]' (D**(2.2) being a hypothetical broad state decaying into Z)7r and -D*vr), and 
have found no combination with a confidence level above 2.7%. We conclude that while these 
modes could contribute to our signal, they are unlikely to account for it fully. 

TABLE II. Detection efficiency for B —* rj'X s modes 



Mode 



B 
B 
B 
B 
B 
B 
B 
B 
B 
B 



#*(892)t/ 

#1(1270)7/ 

#£(1406)7/ 

# 2 * (1429)7/ 

# 3 * (1774) 7/ 
K% (2200)7/ 
D°r]' 
D*°r] f 
D(2.2)t/ 



0.076 ± 0.006 
0.058 ± 0.005 
0.050 ± 0.005 
0.053 ± 0.005 
0.051 ± 0.005 
0.046 ± 0.005 
0.046 ± 0.005 
0.025 ± 0.002 
0.026 ± 0.002 
0.011 ±0.003 



Equal mixture of exclusive b — > sg* modes 
B — > rj'sd, r/'su (JETSET hadronization) 



0.055 ± 0.003 
0.055 ± 0.003 



Finally, we consider b — > sg* . Conventional b — > sqq operators predict an X s mass distribution 
that peaks near 1.5 GeV. This description fits the M(X S ) spectrum poorly (1% C.L.). However, 
the process b — > sg* with g* — > 377' from the QCD anomaly, which has the attractive feature that 
it accounts for the large mass of the 7/ relative to other members of its SU(3) multiplet, gives a 
three-body gsq mass spectrum that peaks above 2 GeV p), 1 1C ] — [13| as shown in Fig. 2. A fit of 



7 



10.0 



3120298-001 



5.0 



2.5 



E 



1.30 



2.30 



1.55 1.80 2.05 

M (pseudo - D) (GeV) 

FIG. 4. The continuum-subtracted distribution of pseudo- D mass (solid) with the expectation 
from B° — > D*°rj' (dashed). The simulation has been normalized to the data yield. 



this model to the data gives a C.L. in the 25 - 72% range. 

In what follows, we shall compute the B — * X s rf branching fraction assuming that it is due 
to b — > sg*, and allow for a background from £)(*(*)) 7/ assuming that these decays occur at rates 
consistent with standard expectations. It is also possible, however, that £)(*(*)) rj is occuring at 
a rate that is up to five times larger than expectation; if so, half our signal could be from these 
modes. 

The detection efficiency for b — > sg* and for the color-suppressed b — > c decays are listed in 
Table II. We observe that the efficiency for the b — > c decay mechanisms is half that for b — > sg* , so 
the computed B — > X s r]' branching fraction depends on our assumption that the signal is 6 — > sg* . 
The b — > sg* decays are modeled by allowing the JETSET Monte Carlo to hadronize the s quark 
and gluon. We also generate a number of exclusive b — ► sg* decay modes. The average efficiency 
for exclusive modes with equal weights is equal to the JETSET efficiency. The detection efficiency 
is averaged over B° and B + mesons and corrects for unobserved modes with K° mesons but does 
not include rj branching fractions. To determine the uncertainty in efficiency due to the modeling 
of the signal, we vary the relative weights of different modes (increasing the fractions of K^rj and 
K\rj to 50%); this leads to a systematic error of 16%. No attempt has been made to calculate 
the branching fraction for decays that lie outside the rj momentum window, as such a calculation 
would be extremely model dependent. 

The dominant source of experimental systematic error is due to the modeling of the X s 
system [14|. Other sources include the choice of background parameterization and the uncer- 
tainty in the tracking and photon detection. We have also included a second systematic er- 
ror for the possible contribution of color suppressed b — > c modes. This is determined by us- 
ing the largest model prediction for these modes and taking into account their lower accep- 
tance. The theoretical predictions are multiplied by 1.5 as an estimate of the theoretical un- 
certainty. Assuming an average detection efficiency of 5.5%, appropriate for b — > sg*, we obtain 
B(B -> ri'Xs) = (6.2 ± 1.6(stat) ± 1.3(sys)±f%(bkg)) x 10" 4 for 2.0 < jty < 2.7 GeV/c. 

A number of interpretations have been proposed to account for the large branching fraction 
of B — > rj'X s . These include: (I) conventional b — > sqq operators with constructive interference 
between the uu, dd, and ss components of the rj p5|,p^|, (II) b —>■ ccs decays enhanced by cc content 
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in the rj wavefunction [fT^Jlgfl , and (III) b — » sg*,g* — > grj' from the rj QCD anomaly ||, [10|- [13]. 
The observed branching fraction is larger than what is expected from Scenario I. Furthermore, 
scenarios I and II will give an X s mass distribution peaked near 1.5 GeV. Only scenario III gives 
a three-body gsq X s mass spectrum that peaks above 2 GeV. 

We have also searched for high momentum B — > rjX s decays, with rj — > 77 and r/ — > it~it + it 
and 2.1 < < 2.7 GeV. In the 77 —* 77 mode, we observe 164 ± 17 events on resonance, 50 =L 10 
events below resonance. The expected b — > c background is 14 ± 8 events. This gives a net yield 
of 54 ± 26 events. In the rj — > 7r~7r + 7r mode, we obtain an on-resonance yield of 52.4 ± 10.5 and 
a below-resonance yield of 24.9 ± 6.7 events. This corresponds to an excess of 4.9 ± 16.5 events. 
The limit obtained by combining the two rj decay modes and allowing for systematic uncertainty is 
B(B — » ?]V S ) < 4.4 x 10 -4 . The theoretical expectation is that this rate will be suppressed relative 
to B — > r]'X s by tan 2 Op |5|,^] where #p is the pseudoscalar mixing angle, which is consistent with 
our result. 

In summary, we have observed a signal of 39.0 ± 11.6 events in high momentum B — ► 77' X s 
production. The interpretation 6 — ► sg* is consistent with all features in the data. Given that 
interpretation, the branching fraction is B(B — > rj'X s ) = (6.2±1.6(stat)±1.3(sys)^i 2 l (bkg)) x 10 -4 
for 2.0 < jty < 2.7 GeV/c. 

We gratefully acknowledge the effort of the CESR staff in providing us with excellent luminosity 
and running conditions. This work was supported by the National Science Foundation, the U.S. 
Department of Energy, Research Corporation, the Natural Sciences and Engineering Research 
Council of Canada, the A. P. Sloan Foundation, and the Swiss National Science Foundation. 
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